Module 13: Local Optimization Algorithms

Homework

Homework 7 is due in two weeks, Sunday at midnight

Learning Objectives

• Neural Network Learning
• Gradient Descent
• Convergence Rates and Condition Number
• Acceleration Using Momentum

Readings

• Chapter 8.1 to 8.4 of Intro to Optimization Algorithms
• Wikipedia article on SGD Covers everything but good level
• Pankaj Mehta’s Notebook I followed his notebook partially here
• Deep Learning Book 8.3 Parts of this chapter are very advanced
• Gilbert Strang’s Linear Algebra and Learning from Data VI.4 and VI.5 Advanced and Scatterbrained, covers too much, but has deep insights
• Accompanying Lectures 22-26 on this playlist